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Preface 


This  report  is  one  of  a series  dealing  with  the  critical  performanc 
requireinents  for  different  occupational  groups. 

The  author  gratefully  acknowledges  the  subject  matter  advice  and' 
technical  assistance  of  the  following  men  in  conducting  the  analyses: 

John  H.  Adams,  Jr.,  Office  of  the  Inspector  General 

Glen  A.  Fast,  Office  of  the  Inspector  General 

The  technical  aspects  of  this  study  are  contained  in  technical 
appendices  which  are  available  from  the  research  files  upon  request. 
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SUMMARY  AND  HIGHLIGHTS 


This  report  describes  the  critical  incident  method  of  determining 
the  performance  requirements  for  auditors  and  special  agents  and  for 
developing  a. personnel  performance  information  system  for  these 
occupations.  The  advantages  of  using  performance  records  as  source 
documents  for  many  kinds  of  personnel  actions  are  explained.  Various 
ways  in  which  management  can  use  the  information  contained  in  pools  of 
critical  incidents  to  aid  decision-making  in  such  areas  as  career 
programming,  spotting  training  needs,  evaluating  the  relevance  of 
training  courses,  and  appraising  the  effects  of  program  and  policy 
changes  are  also  illustrated. 


Chief,  Personnel  Research  Staff 
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To  answer  the  question,  *'How  are  we  doing,**  management  depends  upon 
the  adequacy  of  its  performance  information  system.  In  any  occupational 
area,  if  we  have  inadequate  information  about  the  requirements  for  satis- 
factory performance  and  the  achievement  of  these  requirements,  we  cannot 
tell  how  well  we  are  doing. 

During  the  past  three  years  the  Personnel  Research  Staff  of  the 
U.  S.  Department  of  Agriculture  has  been  doing  research  aimed  at  improv- 
ing our  performance  information  systems.  The  following  is  a study  of 
critical  performance  requirements  for  the  Audit  and  Investigation  personnel 
of  the  Office  of  the  Inspector  General  (OIG). 

We  started  with  the  question:  **What  are  the  critical  requirements 
of  performance  in  this  occupation?**  We  got  the  facts  by  going  to  the 
people  directly  involved  in  doing  and  in  supervising  the  work.  We  asked 
each  of  them  to  describe  cri tical  incidents  that  they  had  observed. 

What  are  critical  incidents? 

Critical  incidents  are  specific  actions  or  behaviors  that  have 
actually  been  observed.  They  do  not  involve  personality  labels  or  lists 
of  traits.  An  effective  critical  incident  leads  to  significantly  better 
than  average  accomplishment  of  a particular  aspect  of  a job,  assignment, 
or  responsibility.  An  ineffective  incident  leads  to  significant  delay, 
mistakes,  omissions,  or  lack  of  accomplishment.  One  might  say  that  an 
incident  is  called  critical  because  it  makes  a real  difference.  The 
judgments  are  based  upon  results--the  outcomes  of  peoples*  actions  and 
behaviors  and  the  products  of  their  work.  They  are  judgments  made  by 
people  at  various  levels,  but  always  by  the  people  in  the  best  position 
to  observe:  those  closest  to  the  operational  situation,  those  best 
situated  to  base  their  judgments  upon  the  objective  facts,  within  human 
limits  of  perceptual  fallibility. 

An  individual  critical  incident  is  not  an  evaluation  of  a person. 

It  is  an  observation  of  **what  happened,**  what  action  took  place,  and 
what  its  consequences  were. 

Each  person  was  asked  to  write  up  eight  incidents--two  effective  and 
two  ineffective  incidents  involving  **job  or  technical  competence**  and  two 
of  each  kind  involving  **working  with  people.**  Altogether  we  collected 
about  3000  critical  incidents  from  Audit  and  Investigation  personnel. 

To  help  in  carrying  out  statistical  studies  of  group  data;  we  also  asked 
for  certain  identifying  information,  but  complete  individual  anonymity 
was  assured. 
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Determining  cri 1 1 ; 


After  the  incidents  had  been  collecteds  vie  sorted  them  into  groups, 
according  to  the  different  critical  performance  requirements  that  they 
reflected.  There  were  no  predetermined  categories.  Sixty-nine  critical 
requirements  grew  out  of  the  content  of  the  Incident  pool.  There  were 
slightly  more  of  these  requirements  for  the  ineffective  than  for  the 
effective  categories. 

The  related  critical  requirements  were  then  grouped  together  to 
identify  eight  critical  performance  areas , We  tested  our  classification 
for  objectivity  by  having  two  observers  independently  sort  a sample  of 
incidents.  The  agreement  among  observers  was  about  747,,  which  was  deemed 
sufficient  for  the  purposes  of  the  study.  This  procedure  yielded  eight 
performance  areas. 

The  performance  record 

A copy  of  the  form  developed  for  Audit  and  Investigation  personnel 
in  OIG  is  presented  in  Exhibit  1,  It  is  organized  by  performance  areas. 
Within  each  performance  area  the  related  critical  requirements  are 
listed““ef f ective  behaviors  on  the  left  side,  ineffective  on  the  right. 
Spaces  are  provided  under  each  performance  area  for  the  supervisor  to 
make  notes  of  critical  incidents  at  the  time  they  occur.  He  will  have  a 
Performance  Record  for  each  of  his  subordinates. 

Experience  in  industry  has  shown  that  on  the  average,  supervisors  of 
groups  of  up  to  thirty  employees  need  no  more  than  five  minutes  a day  to 
make  entries  for  the  whole  group.  This  is  so  for  two  reasons.  First, 
most  of  the  things  that  employees  do  fall  within  the  broad  range  of 
normal,  average,  typical,  satisfactory,  or  acceptable  performance. 
Critical  incidents  that  "make  a real  difference"  are  rarer.  Just  a few 
employees  are  likely  to  be  involved  on  any  given  day.  Some  days  there 
may  be  none.  Secondly,  most  of  the  basic  descriptive  material  is  already 
in  the  Performance  Record,  A short  note  in  the  appropriate  performance 
area  is  all  that  is  needed  to  insure  that  the  information  will  not  be 
forgotten  and  the  incident  can  be  recalled  later  on. 

Suppose,  for  example,  an  Auditor  failed  to  include  in  his  workpaper 
sufficient  information  to  support  his  findings.  Consequently  the  papers 
had  to  be  returned  to  the  Auditor  for  the  additional  information  and 
reviewer  time  was  lost.  Such  an  incident  falls  in  performance  area  1_, 
"Performing  Technical  Aspects  of  Assignments,"  Within  that  area  the 
incident  fits  under  d,  on  the  right  hand  side  of  the  page.  As  to  "What 
Happened,"  the  supervisor  might  merely  note  "No  workpaper  detail  on  audit 
X,  returned,"  which  should  be  enough  to  recall  the  incident  to  his  mind 
at  some  later  time. 
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Exhibit  1 


I.  CARRYING  OUT  ASSIGNMENTS 

1.  PERFORMING  TECHNICAL  ASPECTS  OF  ASSIGNMENTS 


A.  Recognized  inadequacy  in  pro- 
cedures. B.  Detected  impropriety 
not  previously  disclosed  by  audit. 
C.  Noted  peculiarity  and  followed 
through.  D.  Followed  through  on 
finding.  E.  Provided  complete  yet 
concise  support  for  findings. 

F.  Accomplished  assignment  with 
limited  experience  or  guidance, 

G.  Obtained  assistance  when 
needed.  H.  Wrote  report 
exceptionally  well. 


a.  Did  not  make  proper  verification 
(factual  or  numerical).  b.  Did  not 
detect  and/or  follow  through  on 
finding.  c.  Did  not  properly 
select  sample  or  extend  test, 
d.  Did  not  properly  write  up 
support  of  findings  or  make 
adequate  notes,  e.  Did  not  properly 
organize  work  papers  and/or  report, 
f.  Did  not  follow  proper  techniques 
of  interview.  g.  Did  not  request 
assistance  when  needed. 


DATE 

ITEM 

WHAT  HAPPENED 

DATE 

ITEM 

WHAT  HAPPENED 

3/8 

d 

No  workpaper 

detail  on  audit 

X,  returned 

2.  ASSUMING  RESPONSIBILITY  AND  RESPONDING  TO  NEED  FOR  EXTRA  EFFORT 


A.  Thoroughly  prepared  for 
assignment.  B.  Initiated  steps 
to  improve  on- job  performance. 

C.  Went  beyond  scope  of  normal 
responsibilities  (went  into 
greater  detail).  D,  Devised 
short-cut  procedure  for  obtain- 
ing information.  E.  Devised  new 
or  simplified  procedure. 

F.  Demonstrated  alertness  and/or 
follow-through  in  unusual 
circumstances . 


a.  Did  not  adequately  prepare  for 
assignment,  b.  Lacked  knowledge 
of  program  procedures.  c.  Did  not 
recognize  essentials  of  assignment, 
d.  Did  not  recognize  significance 
of  information  obtained.  e.  Did 
not  report  significant  finding, 
f.  Used  too  much  time  in  prepara- 
tion of  work  papers.  g.  Did  not 
maintain  proper  personal  appearance. 


DATE 

ITEM 

WHAT  HAPPENED 

DATE 

ITEM 

WHAT  HAPPENED 
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II o WORKING  WITH  OTHERS 
3o  RELATIONSHIP  WITH  AUDITEES  AND  INTERVIEWEES 


A,  Gained  confidence  or  cooper- 
ation, B„  Extended  consideration, 

Co  Overcame  hostility,  D.  Obtained 
good  acceptance  of  findings, 

Eo  Explained  purpose  of  procedures. 

F.  Pursued  assignment  under  adverse 
condi tions . 

a.  Did  not  ga 
not  extend  coi 
discourteous 
d.  Alienated 
not  adequate!’ 
f.  Became  per 
personnel,  g 
casual  or  too 

in  confidence,  b.  Did 
isideration,  c.  Was 

Dr  argumentative. 

Dr  antagonized,  e.  Did 
/ explain  findings, 
sonally  involved  with 
. Was  too  friendly  and 
aloof. 

DATE 

ITEM 

WHAT  HAPPENED 

DATE 

ITEM 

WHAT  HAPPENED 

7/1 

D 

Audit  Z findings 

presented  so  that 

auditee  was  coopera- 

tive  and  interested 

in  following  through 

4o  COOPERATING  AND  GETTING  ALONG  WITH  OTHERS 


A,  Coordinated  with  another 
individual  or  group,  B.  Remained 
pleasant  and  tactful  under 
provocation. 

a.  Did  not  coordinate  with  another 
individual  or  group,  b.  Argued 
with  or  antagonized  a fellow 
employee,  c.  Made  inadequate 
group  presentation. 

DATE 

ITEM 

WHAT  HAPPENED 

DATE 

ITEM 

WHAT  HAPPENED 



IIIo  MANAGING  WORK  AND  SUPERVISING  OTHERS 
5.  PLANNING  AND  ORGANIZING  WORK 


A.  Organized  for  efficient  use  I Did  not  delegate  responsibility, 

of  personnel,  b.  Did  not  properly  plan  for 

requirements  of  situation. 


DATE 

ITEM 

WHAT  HAPPENED 

DATE 

ITEM 

WHAT  HAPPENED 

— 
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6„  MOTIVATING  SUBORDINATES 

A.  Gave  recognition  for  superior 
skill  or  job  performance.  B. 
Criticism  was  specific  and  con- 
structive. C.  Extended  considera 
tion  for  employee’s  personal 
problems  or  feelings.  D.  Gave 
encouragement  by  showing  interest 
in  employee’s  work  or  welfare. 


a.  Did  not  discuss  subordinate’s 
performance  with  him.  b.  Was  not 
constructive  or  was  unduly  severe 
with  criticism,  c.  Did  not  extend 
consideration  to  employee’s  personal 
problems,  d.  Acted  in  arbitrary  or 
dictatorial  manner.  e.  Criticized 
in  presence  of  others. 


WHAT  HAPPENED 


DATE 


ITEM 


WHAT  HAPPENED 


DATE 


ITEM 


7.  TRAINING  AND  DEVELOPING  SUBORDINATES 


A,  Provided  information  when 
needed.  B.  Gave  help  or  train- 
ing when  needed.  C,  Explained 


a.  Did  not  provide  guidance  or 
training  when  needed.  b.  Did  not 
explain  specific  requirements  of 


specific  requirements  of  job. 

job. 

DATE 

ITEM 

WHAT  HAPPENED 

DATE 

ITEM 

WHAT  HAPPENED 

8.  MAINTAINING  COMMUNICATIONS 


A.  Explained  reason  for 
decision  affecting  employee. 

a.  Did  not  explain  reason  for 
decision  affecting  employee. 

b.  Did  not  clarify  or  answer  ques- 
tions, c.  Did  not  provide  informa- 
tion when  needed.  d.  Gave  improper 
and/or  incomplete  instructions. 

e.  Did  not  give  subordinate  an 
opportunity  to  explain. 

DATE 

ITEM 

WHAT  HAPPENED 

DATE 

ITEM 

WHAT  HAPPENED 
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Again,  suppose  that  the  employee *s  performance  record  showed  the  fol- 
lowing entry  under  22*  "Audit  Z findings  presented  so  that  auditee  was 
cooperative  and  interested  in  following  througho”  The  statement  of  critical 
requirement  3^  will  serve  to  remind  the  supervisor  that  in  this  incident 
the  Auditor  sensed  that  the  auditee  was  nervous  and  disagreeable  about  the 
exit  interview  and  hence  phrased  the  findings  in  a manner  that  was  effec- 
tive but  non-off ensiveo  As  a result  the  auditee  was  cooperative  and 
interested  in  taking  corrective  action  on  the  findings o In  both  examples 
the  relevant  information  is  neatly  summarized  in  a single  short  phrase. 


PUTTING  CRITICAL  INCIDENTS  TO  WORK 


Personnel  performance  information  for  decision-making 


Up  to  this  point  we  have  concentrated  upon  providing  an  understand- 
ing of  what  critical  incidents  are,  and  how  they  are  used  to  define  the 
critical  performance  requirements  of  an  occupation  and  to  develop  a 
performance  record.  Hopefully,  experienced  managers  among  our  readers 
have  begun  to  perceive  the  applicability  and  utility  of  such  information 
at  many  points  across  a broad  range  of  management  activities. 

If  we  consistently  assemble  and  maintain  a store  of  information  on 
individuals*  performance  at  work,  we  will  have  available  the  essential 
facts  needed  to  arrive  at  any  of  the  administrative  personnel  decisions 
we  are  called  upon  to  make.  The  summation  of  the  pluses  and  minuses  of 
individual  performance  can  give  us‘  an  improved  comprehension  of  the 
requirements  that  most  influence  success  and  failure  in  achieving  our 
objectives.  We  will  reduce  the  burden  of  separately  maintaining  large 
overlapping  files  of  performance  information  for  each  administrative 
application  (e.g.  awards,  transfers,  promotions,  annual  ratings,  training 
needs) . 

From  here  on  we  will  explore  in  greater  depth  some  specific  ways 
supervisors  and  managers  can  make  use  of  the  critical  incident  technique 
and  critical  incident  records. 

What  can  critical  incidents  tell  management? 

Our  answers  can  be  developed  in  two  areas:  (1)  individual  supervisor- 
subordinate  relations;  and  (2)  general  personnel  management  functions  and 
programs. 
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Individual  supervi sor-subordinate  relations 

Accumulating  information  about  performance  in  the  critical  require- 
ments of  work  continually  rather  than  intermittently,  provides  a more 
objective  record  of  information.  The  pressures  to  "mold”  the  information 
are  less  than  when  it  must  be  pulled  from  your  head  only  when  a report 
on  an  individual  is  needed  for  some  specific  purpose  This  is  so, 
because  the  ends  to  be  served  by  the  report  tend  to  determine  what 
information  will  be  retrieved  and  offered  on  that  occasion.  Contrary  to 
good  theory,  the  supervisor  in  practice  usually  makes  his  judgment  first, 
and  then  finds  the  ’’facts”  to  support  it,  instead  of  the  other  way 
around.  So,  if  all  performance  information  is  present  as  part  of  admin- 
istrative reports , there  remains  no  independent  source  of  data  collected 
under  ’’neutral”  conditions. 

A supervisor’s  note  in  a person’s  performance  record  is  neither  a 
punishment  nor  a reward.  The  performance  record  is  not  an  evaluation 
form,  nor  is  the  supervisor  attempting  to  evaluate  the  person  when  he 
makes  a note  for  the  record.  He  is  only  evaluating  the  quality  of  a 
specific  action  or  performance.  The  notes  are  merely  reminders  of  facts 
to  be  considered  in  talking  to  the  employee  about  his  work,  and  for  refer- 
ence when  personnel  decisions  are  to  be  taken  later  on.  They  are  as  much 
the  property  of  the  employee  as  of  the  supervisor.  If  the  employee  wants 
to  use  these  notes  to  improve  himself,  they  are  available  for  discussion 
with  his  supervisor.  The  employee  makes  the  record.  The  supervisor  only 
keeps  it. 

Since  observations  are  noted  when  they  happen,  the  information  is 
less  subject  to  forgetting  and  errors  of  recall.  Since  the  critical 
incidents  are  identified  with  the  normal  work  situation  and  discussed 
when  they  happen,  both  parties  are  likely  to  develop  a better  balanced 
perspective. 

As  a consequence,  the  significance  of  individual  incidents  is  less 
likely  to  be  inflated  out  of  all  proportion  to  their  place  in  the  total 
picture  of  performance. 

Because  the  incidents  deal  with  observable  behavior  and  its  results, 
it  is  possible  to  discuss  performance,  and  even  to  evaluate  it  on  a less 
emotion-provoking  level  than  is  so  often  the  case  when  inability  to 
recall  specific  illustrative  incidents  leads  to  generalized  and 
questionable  personality  analysis. 

Furthermore,  the  summaries  of  an  individual’s  performance  history 
that  underlie  the  evaluations  and  decisions  needed  in  performance  review, 
career  planning,  promotion  rosters,  and  training  plans,  are  less  likely 
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to  be  looked  upon  as  bureaucratic  procedures,  largely  divorced  from  the 
realities  of  day-to-day  experience  and  the  normal  relations  between  a 
man  and  his  boss.  This  can  mean  improvement  not  only  in  job  performance, 
attitudes,  and  common  understanding,  but  in  the  auality  of  supervision 
as  well. 

In  these  circumstances,  forms  and  procedures  are  developed  to  facili- 
tate the  use  of  information,  but  the  forms  are  clearly  subordinated  to 
the  substance  of  the  information  systemo 

As  a result,  the  occasions  when  formal  review  and  evaluation  take 
place  are  less  likely  to  be  regarded  as  uncomfortable  special  occasions 
during  which  both  parties  must  communicate  self-consciously,  with  their 
guards  up.  The  "summing  up”  is  more  likely  to  be  looked  upon  in  a 
healthy  fashion  as  the  natural  consequence  of  shared  experience,  and  as 
part  of  a jointly  held  aim  to  further  the  success  of  the  outfit  of  which 
both  are  a part. 


General  personnel  management  function  and 


Apart  from  those  situations  of  an  individual  nature,  the  pools  of 
critical  incidents  that  are  accumulated  offer  valuable  knowledge  to 
managers.  Given  supplementary  information  on  such  characteristics  of 
the  groups  involved  as  grade  level,  length  of  service,  age,  sex,  and 
occupational  specialty,  useful  comparisons  can  be  made..  Again  let  us 
look  at  some  examples. 


Taking  the  pulse  of  the  organization 

What  are  the  areas  of  particular  strength  or  weakness  in  the 
functioning  of  the  organization  as  a whole?  Consider  Exhibit  2.  Exhibit 
2 is  based  upon  a sample  of  about  1500  incidents  drawn  from  the  pool 
of  3000.  We  see  that  relatively  high  concentrations  of  both  effective 
and  ineffective  incidents  occur  in  the  areas  of  "Relationships  With 
Audi  tees  and  Interviewees,"  "Performing  Technical  Aspects  of  Assignments” 
and  "Assuming  Responsibility  and  Responding  to  Need  for  Extra  Effort,” 
However,  "Performing  Technical  Aspects  of  Assignments,"  shows  a higher 
concentration  of  ineffective  than  effective  incidents,  while  the 
reverse  is  true  of  "Assuming  Responsibility  and  Responding  to  Need  for 
Extra  Effort"  and  "Relationships  With  Auditees  and  Interviewees."  The 
area  of  "Motivating  Subordinates"  also  has  a high  concentration  of 
effective  and  ineffective  incidents.  On  the  other  hand,  relatively  few 
incidents  of  either  type  occur  in  the  area  of  "Planning  and  Organizing 
Work . " 


PERCENT  OF  EFFECTIVE  AND  INEFFECTIVE  INCIDENTS  BY  PERFORMANCE  AREAS 
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The  significance  for  management  of  a tabulation  like  this  is 
obvious.  It  pinpoints  the  areas  where  further  analysis  might  be  profit- 
able. Take  just  one  example.  A comparison  of  the  distribution  of 
critical  incidents  with  present  statements  of  job  requirements  can  show 
whether  or  not  the  latter  need  to  be  revised. 

Making  promotion  decisions 

How  do  the  critical  requirements  of  the  job  change  with  grade 
level?  Exhibit  3 shows  the  proportions  of  incidents  at  different  grade 
levels  for  each  performance  area.  At  the  lower  levels,  the  greatest 
proportions  of  incidents  involve  "Performing  Technical  Aspects  of 
Assignments,"  "Assuming  Responsibility  and  Responding  to  Need  for  Extra 
Effort,"  and  "Relationships  With  Auditees  and  Interviewees."  At  the 
higher  levels,  "Motivating  Subordinates,"  "Training  and  Developing 
Subordinates,"  and  "Maintaining  Communications"  account  for  a greater 
proportion  of  incidents  than  at  the  lower  levels.  At  each  grade  level, 
"Relationships  With  Auditees  and  Interviewees"  remains  relatively  high 
in  its  proportion  of  incidents.  No  surprise  here.  Nevertheless,  an 
analysis  of  critical  incidents  can  help  to  define  the  content  and  changes 
in  work  that  occur  at  these  different  levels,  thus  providing  management 
with  more  specific  information  than  might  otherwise  be  available. 

When  incidents  are  classified  and  tabulated  as  in  Exhibit  3,  it 
quickly  appears  that  the  supervisor  may  not  have  the  opportunity  to 
observe  behaviors  at  one  level  that  are  indicative  of  how  well  the 
employee  will  perform  at  a higher  level,  because  there  may  be  little  or 
no  opportunity  for  these  kinds  of  incidents  to  occur.  As  Exhibit  3 
clearly  shows,  "Planning  and Supervising"  incidents,  which  preponderate 
at  the  higher  grade  levels  occur  very  infrequently  at  the  lower  levels. 
Hence,  in  making  a selection  for  a supervisory  job,  a promotion  board 
might  wish  to  supplement  performance  information  with  other  data  - for 
example,  tests  - to  get  a better  prediction  of  how  well  a person  might 
be  expected  to  function  as  a supervisor. 

Planning  job  rotation 

A job  rotation  plan  can  be  developed  by  classifying  the  incidents 
according  to  functional  areas.  This  would  enable  participants  in  the 
plan  to  gain  experience  in  certain  kinds  of  problem  situations  that 
would  enhance  their  performance  later  in  their  careers. 

Spotting  training  needs 

Just  as  an  individual  employee’s  perfbrmance  record  can  be  used  to 
diagnose  his  specific  training  hdeds,  so  too  can  a pool  of  critical 
incidents  collected  from  many  employees  point  up  training  needs  of  the 
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group  as  a whole.  This  is  particularly  true  when  the  incidents  are 
classified  according  to  function,  or  program  areas.  Inspection  of  the 
incidents  may  suggest  curriculum  content,  as  for  example,  certain 
supervisory  practices  that  need  to  be  emphasized  or  .re-emphasized. 

Indeed  the  incidents  themselves  might  be  incorporated  as  study  material 
in  a supervisory  training  program. 

Determining  relevance  of  training 

Critical  incidents  can  also  be  used  to  assess  and  validate  exist- 
ing curriculum  content  by  checking  them  as  to  whether  the  relevant 
skills  are  ’’Taught"  or  ’’Not  Taught’’  in  the  course. 

Evaluating  program  changes 

Still  another  potential  use  for  critical  incidents  is  the  evalua- 
tion of  program  and  policy  changes.  Suppose  that  a certain  organizational 
or  procedural  change  is  introduced  in  order  to  alleviate  some  problem 
situation.  Incidents  collected  after  the  change  has  been  introduced  can 
be  compared  with  those  in  the  present  incident  pool,  and  a statement  as 
to  the  effects  of  the  change  then  can  be  made. 


CONCLUSION 


It  should  be  evident  by  now  that  critical  incidents  can  tell  manage- 
ment many  things  that  it  needs  to  know.  The  information  derivable  from 
an  analysis  of  critical  incidents  is  less  influenced  by  subjectivity  and 
bias,  because  it  is  based  upon  actually  observed  job  performance.  At 
the  same  time,  it  is  more  precise,  more  specific,  and  more  relevant  to 
the  requirements  of  the  job  than  information  obtained  by  more  traditional 
methods. 
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